Named Entity Recognition and Disambiguation in Tweets Master Thesis

Social media has grown exponentially over the past few years. Users are generating far more unstructured content than ever before. Successful companies are also very active in social media analysing these data for their marketing campaigns. But the informal and noisy nature of such data makes it quite difficult to extract meaningful information out of them. In this thesis, we investigate the problem of named entity recognition and disambiguation in tweets. Given a tweet written in English, the task focuses on extracting all the named entities inside and assigning each entity with a reference link. It has potential applications in online product monitoring, sentiment analysis, electoral predictions and other social media analytics. We present a five step approach for this problem consisting of tokenization, Part-of-speech tagging, normalization, mention extraction and disambiguation. First three steps are used as preprocessing steps for mention extraction and disambiguation which are the core components of our approach. For tokenization and partof-speech tagging, we use the regular expression tokenizer and tagger developed by Owoputi et al. In the third step, we propose a normalization algorithm which uses Brown clusters and Microsoft web ngram language model to normalize the tweets. In the next step, our mention extractor extracts possible candidate entities with the help of POS tags and ngram matching against Freebase. Finally, using the intra-tweet context and relationship among entities, our disambiguation algorithm assigns appropriate reference links to these extracted entities. To evaluate our approach, We carry out several experiments and benchmark our complete process against different state of the art extraction and disambiguation systems. We also evaluate the importance of individual steps by measuring the overall performance degradation by leaving that step out of the pipeline. Finally, we present a case study demonstrating how our approach can be applied practically in a real world scenario.